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VIA A TELEPHONE 
(57) Abstract 

Voice activation of function! on a networic «ueh as the 
Internet are accomplished using a speech recognition system 
running synchronously wish standard desktop-based teternet 
ftnetions. This synchronous operation allowa voice-based 
contsol to be exercised for all operations on the Internet, System 
functions are based on a unique combination of a local web 
browser, a remottly-tacated speech/web server, and control links 
between a web browser and a speech/web server. The control 
links provide a mcchanisni for controlling a speech server from 
a web page and a mechanism for driving both the local, as well 
as a remote, web browser. 
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METFTOn AND SYSTEM FOtt miNfl KPrnrtf 
KECQGNTTTQN TO ACCESS THE INTER NET TNCTlTnTNO 
ACCESS VTA A TRI Fp^fffffi 

HELD OF THE INVENT! 

The present invention relates to the field of computerized communication on 
the Internet. The general purpose of this invention is to enaWc speech access to the 
Internet over standard telephone lines and Internet control of telephony functions 
through standard web pages. This is accomplished through a unique combination of 
speech server, web browser, and control links. The control links provide a mechanism 
for controlling the speech server from a web page and a mechanism for driving both 
the local, as well as a remote! web browser. 
BACKGROUND OF THE INVENTION 
15 The Internet is essentially a network of servers containing information that 

users can obtain using personal computers. Users generally connect to a server, a 
computer equipped with inforrnation and capabilities that assist the user with 
contacting other servers and obtaining additional information. Users typically execute 
these functions, also referred to as "navigating" on the Internet, using a mouse and 
20 Windows-based software. The user's navigation of the Internet is thus essentially 

graphically-based (looking at a screen) with functions activated using a mouse. 

Speech recognition software and hardware for use in conjunction with personal 
computers and other environments, like the Internet, is a rapidly developing 
technology. With speech recognition, a user's voice commands are recognized by a 
computer and then converted, based on the speech pattern, into an electronic signal. 
For example, speech recognition has been highly successful in the field of long-distance 
telephone calling for the purpose of allowing collect calls. Typically, with this 
application, a caller will provide a name and a phone number to a computer when 
making a collect call. The computer will then place the caller on hold and call the 
number to be reached- The person receiving the collect call will answer "yes" or "no" 



30 
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in response to the computer message and the collect caller's name. The voice 
recognition hardware and software, which is also known aa a speech recognition 
engine, either signals a switch to complete the call upon recognizing the "yes" 
response, or to disconnect upon recognizing the "no" response. 
5 One issue with using speech recognition is selecting the appropriate speech 

recognition engine to use for a particular application. These speech recognition 
engines include speaker dependent and independent dictation machines; continuous 
speech systems, large vocabulary systems, and small vocabulary systems. Further, 
these systems can be Windows based, Macintosh based, UNIX based, Windows NT 

10 based, or based on another platform, depending on the preferred operating system. 

Speech recognition operating in conjunction with computer connection with 
the Internet, also known as speech enabling of the Internet, appears to have promising 
application possibilities. One possible application of this technology is for navigational 
purposes on the Internet For example, speech recognition has been successfully 

13 utilized at the desktop level generally. Voice macros have been created for a number 

of Windows functions for use on the Internet. A macro is a series of functions on the 
computer activated by a single command. For a voice macro, the speech server's 
recognition of an inputted voice command activates a series of commands. 

Two prior art methods for speech-enabling the Internet have been explored by 

20 various companies and research entities. In general terms, researchers have 

approached the problem from either the perspective of speech-enabling the Internet, or 
from the perspective of Internet-enabling the telephone system. 

The first method is the most common approach and the one being pursued by 
Texas Instruments, Apple Computer, and Microsoft. In this approach, the speech 

25 recognition engine is located on the local host, along with the web browser. This 

approach allows such activities as those described above - voice macros for Windows 
functions that can be used when browsing the Internet. 

Texas Instruments further refined this approach by using the text associated 
whh hotlinks to supply the vocabularies for the recognizer. Apple has taken the 

30 approach of making both the web browser and the speech recognition engine script able 
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(controllable with the AppleScript lavage). Microsoft has taken the approach of 
providing tools for web page developers to allow them to speech-enable their web 
pages. These tools provide a mechanism for supplying the recognizer with grammars 
and their speech synthesizers with spoken prompts. 

The advantages of the present invention over this method include: (1) 
telephone access serves a far greater potential audience than speech access limited to 
desktop operations; (2) no additional requirements of the user's computer, such as a 
speech recognition engine, are required; (3) the system uses a migration path starting 
with an immediate utility with no long-term limitations; and (4) direct benefits are 
available from telephony integration. 

Internet-enabling the telephone system is primarily being investigated as a 
research effort Demonstrations from MIT and the Sun SpeediActs group have shown 
potential for using a speech-only interface for retrieving personal information (voice e- 
rmij) over the phone and for using the Internet as an up-to-date repository of 
information available over the phone. For example, ALTech, a commercial spin-off of 
MIT, has demonstrated the use of a speech server for obtaining information about local 
movies. 

Advantages of the present invention over this method include: (1) an optional 
Graphical User Interface (GUI) makes using the system with today's World wide Web 
much more practical and sample than attempting to do it with speech alone; (2) the 
potential user base is just as large over the long term; and (3) providing tools to other 
developers is expected to lead to much more rapid progress than attempting to build 
speech-only interfaces from the ground up. 
SUMMARY QE T HE INVFNTTflM 

This invention links networks such as the Internet and the World Wide Web to 
a speech recognition server, which resides on the telephone system, to provide for 
speech access to these networks over standard telephone lines and control of telephony 
functions through standard web pages. These capabilities are accomplished through a 
combination of speech server (typical of those found in Interactive Voice Response 
(IVR) applications), web browser, and control links. The control links consist of 
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software that provides a mechanism for controlling the speech server from a web page, 
and a mechanism for driving both the local, as well as a remote, web browser. 

An example of the capabilities of the system is as foUows. A user seeking a 
service to provide stock quotes can access these quotes by graphically browsing the 
5 Internet to a web page that continually carries the quotes. Once at the web page, the 

user can activate the present invention, telling the speech server to, for example, "mark 
this" or "show me the stock quote." The server can then be set to either tell the user 
the stock price or go to that web page upon recognizing of the selected speech pattern. 
The general purpose of this invention is thus to provide a method for linking a 

10 remote speech recognition device operating over the telephone network to any web 

browser operating over the Internet. This link enables the user 1 s web browser to be 
controlled by the remote speech recognition device, and, in turn, enables telephony 
functions to be controlled by any web browser. In addition to providing an immediate 
solution to accessing the web by voice, the invention provides tools and motivation for 

15 web page authors to generate web pages that are tailored to speech-only interfaces. 

This is expected to transform the nature of the web, and, over time, to support a truly 
multi-modal interface with the Internet. 

The significance of the invention is that it provides both a means for 
immediately speech-enabling the Internet and a means for gradually Internet-enabling 

20 the telephone system. Other systems have approached the problem of linking speech 

technology and the Internet from either one perspective or the other (that is, speech- 
enabling the net or net-enabling the telephone). The approach of the present invention, 
however, can be viewed from either perspective, and, in so doing, leads to an 
immediate speech-enabling of the Internet, and to a process of Internet-enabling the 

25 telephone. In addition, the present invention leads to functionality completely 

unobtainable from either of the other approaches taken alone. 

The control of both the server's web browser and the user's remote web 
browser also enables an optional GUI for the user of the Speech/web server. The GUI 
link is not required for the system to operate; however, because the web is currently 

30 graphically-oriented, the ability to use the local web browser as a GUI for the speecb- 
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driven browser is expected to be beneficial when surfing the web by voice. The 
concept of a telephony-based web browser with an optional GUI constitutes a 
significant attribute of the system because it provides a common platform that can be 
used for simple applications by anyone with a telephone. In addition, it can be used for 
more difficult tasks when a PC or workstation is available to the user. 

Another example of the use of the present invention pertains to speech input 
and output over telephone fines as the additional modality that can be linked to the 
conventional web browser inter&ce. Thus, rather than placing a call, hanging up, and 
placing another call, a user wiD be able to browse using the telephone. This browsing 
includes such activities as seamlessly speaking to one person, and then connecting to 
another, and then checking messages and ordering a pizza, all without hanging up and 
without ever dialing a number. The same method links any alternative user inter&ce to 
the user's standard web browser. This pertains to browsers with teletypewriter (TTY) 
interfcees, browsers that un demand and speak other languages, or even browsers 
15 capable of providing a sense of smell, sight, taste, and touch. 

Additional objects, advantages and novel features of the invention will be set 
forth in part in the description which follows, and in part will become more apparent to 
those skilled in the art upon examination of the following or may be learned by practice 
of the invention. The objects and advantages of the invention may be realized and 
attained by means of the mstruirietrtalities and convocations particularly pointed out in 
the appended claims. 

To achieve the stated and other objects of the present invention, as embodied 
and described below, the invention may comprise the steps of: 

accessing a voice recognition server through a voice transmission device; 
such device translating voice transmissions into electronic signals; and 
using said translated voice transmissions to perform functions on the Internet 
via voice translation being performed by said server. 
BRIEF nESCRTPTTON OF THE ^f^q^ 

A block diagram of the invention is shown in figure 1 . 
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Figure 2 shows how a user that happens across the web page containing 
connection information on the present invention initiates the process of speech 
enabling his or her web browser using the preferred embodiment. 

Figure 3 illustrates the exchange of information necessary to speech enable a 
5 web browser. 

Figure 4 shows the connections in place for operation of the preferred 
embodiment 

Figure 5 illustrates all of the components of the system in operation. 
Figure 6 contains an alternative embodiment, in which the local web browser is 
10 a slave to the speech/web server. 

Figure 7 contains a second alternative erabodhnent, in which the speech/web 
server is a slave to the local web browser. 
iwr att m nraranmnN of the PREFERRED EMBODIMENT 

Using the drawings, the preferred embodiment of the present invention will 
IS now be explained. 

A block diagram of the invention is shown in Figure 1. A local web browser 1, 
such as Netscape on a PC, is used to browse the Internet 2 using a conventional 
Transport Control Protocol (TCP) fink 3 . The local web browser I contains an 
Appfied Speech Technology Protocol (ASTP) plug-in 4, which communicates by 
20 ASTP link 5 with an ASTP controller 6 located within a speech/web browser 7 of a 

speech/web server 8, such as a Pentium processor-based PC running Windows NT. 
This PC also hosts, or a separate PC coupled to the speech/web server 8 hosts, the 
speech server 9, which is coupled 10 to the ASTP controller 6. These couples can 
consist of such connections as an electronic circuit, a fiber optic line, an 
25 electromagnetic signal, or any other means of coupling known in the art A Dialogic 

fine card located in the backplane of the speech server 9 PC couples 11 the speech 
server 4 to a telephone network 12. The speech/web browser 7 is also TCP linked 13 
to the Internet 2. 
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The three major components of the speech/web server thusare the speeciVweb 
breathe speech »e W 9^ tt ^ a ^ - toA8ff 
software 4 and 6. 

CTD J* ^ 7 » a off-thc^elf web browser with an 

ASTPpfcg.i.,6. m ^P^^*^bdow J s aS oWpro gran , 
wntten B a language, s^h as JAVA, that allows the program to run within a web 
bn^.suchasNetscape. However, since me speeeNweb browser 7 is driven by 
Wm^kk^^^,^ ^givesftaco^derabteresponae 
fme advantage over a brows, that must download and display graphics. The tin, 

compile the grammar for the new web page. 

The speech server 9 i s typicaJ of those used for TVR and operator usist 
appfica^n, These systems vary considerably in the number of sirmaitaneou, channel 
of speech recognition they can support, tu, are moat often Wit from of^he-shelf 
^Wm^p^sKiAT^. Atypicdcon^^fo,^^ 
would be aPentium class PC running UNIX or Windows NT, loaded with a speech 
-ogr^er such as ALTech, PureSpeed, or Nuance, whb a Dialogic line card capable 

^w 1 u, fe urchanneUofrecogriuon. Speed, output is either from pre-recorded 
pnnnptsora speech synthesis. The tdephone line cardea*!* the system, o dial 
out, receive calls, and to conference calls. 

^"^-^.stbehewofthes^ 

wmten in a typical software mat csn operate in Netscape, such as JAVA The 
protocol ft a superset of the Common Client Imerface (CO), which provides the 
mec*™ for estabfchmg a persistem m ^ ^ ^ 

^.browser (local web browser 1). The persistent hnk enables ttespeeduweb 
bowser 7 to remotely control theuser's web browser 1, the user's web brow*, 1 to 
control the speech/web browser 7, and also allows the two browsers 1 and 7 to 
traverse the web in tandem. 
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In addition to the CCI-fike capability, the ASTP protocols provide the interface 
to the speech server 9, telRng the recognizer what grammar to compile for the next 
web page. This function is typically fulfilled by simply stripping the text associated 
with each hottink and sending it to the recognizer's grammar compiler. Alternatively, 
5 versions of the protocol support calls to high-level routines, called "speech behaviors/ 1 

that handle all of the dialog between the user and the machine. These high-level 
routines allow users to supply, by voice, specific kinds of information when using the 
Internet, such as credit card numbers, addresses, and telephone numbers. By providing 
web page authors with access to well-designed dialog modules that can be easily 

10 deployed through simple-to-use web authoring tools, such as the ASTP protocols, the 

predominately graphical nature of the web changes to accommodate a speech-only, 
telephone-based interface. 

Finally, the ASTP link 5 is what provides the conduit between the web page 
and the telephone. This allows web authors to include telephone numbers associated 

15 with hotlinks that can be dialed by the speech/web server 8. This capability may 

change how switching is currently done in the telephone network 12. 

Figure 2 shows how a user that happens across the web page containing 
connection information on the present invention initiates the process of speech 
enabling his or her web browser using the preferred embodiment. A user 15 using a 

20 local web browser 16 initiates a TCP connection 17 with the speech/web site, which is 

served by the speech/web server 18, by selecting a hotttnk such as "surf the web by 
voice" at the web site. 

In Figure 3, user IS of a local web browser 16 and local telephone 19 uploads 
20 to the speech/web server IS from the local web browser 16 the local telephone 

25 number and downloads 21 the ASTP plug-in from the speech/web server 18. In Figure 

4, the user 15 of the local web browser 16 and local telephone 19 simultaneously 
connects by ASTP connection 17 and by telephone connection 22 with the speech/web 
server IB. 

The setup of the preferred embodiment is now completed, as shown in Figure 
30 5. The user 15 of the local web browser 16 and local telephone 19 simultaneously 
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communicates with the speech/web server 18 via ASTP connection 17 and telephone 
connection 22. The user IS is also connected by a TCP link 25 to other web servera 
24 simultaneously 26 with the speech/web server 18 connection by a TCP link 23 with 
those other web servers 24. 

As a result of these simultaneous links 26, the user can browse the Internet 
usmg voice while looking at the screen of the local web browser 16 and speaking over 
the phone ] 9. Typically these links allow a user to speak into the phone using words 
within the system's capability. These words are recognized and interpreted by the 
speech/web browser located at the speech/web server 1 8 and translated into a TCP link 
23 command for the speech/web browser at the speech/web server 18. At the same 
time, the ASTP supplies the same TCP link command 17 on the local web browser 16. 
Thus, the user 15 speaks to control browsing of the Internet. 

A significant advantage of the preferred embodiment is responsiveness. The 
dual Enk approach allows time for the speech/web server to generate grammars while 
the user's browser is busy displaying graphics. A secondary advantage is that neither 
of the web browsers need to be mocBfied for the system to work. 
Variation and Modifications 

Two variations on the invention are illustrated in Figures 6 and 7. These 
approaches differ Bom the one described in Figure 1 in that they require only a single 
fink into the Internet, rather than the two links described previously. 

In the method shown in Figure 6, the local web browser 1 with ASTP plug-in 4 
is linked 5 to an ASTP controller 6 located within a speech/web browser 7 housed 
within a Pentium processor PC-based speech/web server 8. This PC is typically 
ninning Windows. This PC also hosts, or a separate PC coupled to the speech/web 
server 8 hosts, the speech server 9, which is coupled 10 to the ASTP controller 6. The 
speech setver 9 is linked 1 1 to a telephone network 12. The speech/web browser 7 is 
also TCP linked 13 to the Internet 2. 

The primary difference between this alternative and the earlier embodiment 
(Figure 1) is that a direct link 13 does not exist between the speech/web browser 6 and 
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Internet 2 (link 3 of figure l). 

In the method shown in figure 7, the Joel web browser 1 with ASTP plugn. 4 
» imfed 5 to an ASTP control 6 located within a ape**/** ^ 7 housed 
wn^aPem^^r^pc^^^^ This PC also hosts, or a 

aetWDrk12 - The '-1 web browser 1 is also TCP linked 3 to the Internet 2 

ft.ta.i2 0U 13 ofHgure 1) sintuUaneou, with alb* 3 between thelocalweb 
browser 1 and the Interna 2. 
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what rs rrAiyrFpffi. 

1. A method for miliar* speech recognitioii ofvoice commands by a user to 
connect with and perform fimctions on a network via a web site that is served by a 
voice recognition server, comprising the steps of: accessing said voice recognition 

5 server through said web site; accessing said voice recognition server through a voice 

transmissian device; translating voice transmissions into electronic signals; and using 
said translated voice transmissions to perform functions on the network via voice 
translation being performed by said server. 

2. The method of claim 1 whereby said access to the Internet is through a local 
10 web browser. 

3. The method ofclaim 2 whereby said local web browser is a PC. 

4. The method of claim 1 further including the step of said user providing a 
voice connection identification to said voice recognition server. 

5. The method of claim 4 wherein said voice connection identification is a 
15 phone number. 

6. The method of claim 1 whereby said voice transmission device is a 
telephone. 

7. The method of claim 1 whereby said voice transmission device is a 
microphone coupled to a computer. 

* n * tnethod of claim I whereby said access of said voice transmission 
device is through a telephone network. 

9. The method of claim 1 whereby said access is through a telephone line. 

10. The method cf claim I further mcluding the step of said voice recognition 
server instructing said local browser to perform said function. 

11. Theniethodof^lOror^ 
instruction through software loaded on said local browser. 

12. The method of claim 1 1 whereby said software is downloaded from said 
voice recognition server. 

13. The method of claim 10 whereby said instruction is through software 
loaded on said voice recognition server. 



20 



PAGE 33/42 ■ RCVD AT 2/1 5/2005 0:20:46 PM [Eastern Standard Timel ■ SVR:USPTO-EFXRF-1/2 * DNIS:8729306 * CSID:732 530 9808 * DURATION (mm-ss): 1 2<58 



02/15/2005 18:31 FAX 732 530 9808 

WO 99/08084 



MOSER PATTERSON SUER I DAN -+ PTO 



PCT/US98/1552S 



©034/042 



12 

14. A system for utilizing speech recognition of voice commands by a user to 
connect with and perform functions on a network, comprising* local web browser 
coupled to a speech/web server, voice information transmitted from said user to said 
server via a telecommunication connection; a software program for communicating 

5 speech recognition information and executable functions between said local web 

browser and said speech/web server, a speech server for translating said voice 
information into electronic signals for use by said speech/web browser; a speech/web 
browser residing cm said speech/web server for executing functions; a coupling 
between said local web browser and said speech/web server such that said speech/web 

1 0 server executes functions on said local web browser in response to voice commands; 

and a coupling between said speech/web server and said network. 

15. The system of claim 14 in which said speech/web browser includes 
software for navigating the Internet 

1 6. The system of claim 14 in which said speech/web server consists of a 
15 personal computer. 

17. The system of claim 15 in which said speech/web browser runs within said 
speech/web server. 

1 8. The system of claim 14 in which said software program for communicating 
said speech recognition information and said executable functions operate within said 

20 speech/web browser. 

19. The system of claim 14 in which said local web browser includes software 
for navigating the Internet. 

20. The system of claim 19 in which said local web browser runs on a personal 
computer. 

25 21. The system of claim 14 in which said software program for communicating 

said speech recognition information and said executable functions operate within said 

local web browser. 

22. The system of claim 14 in which said tdecommunication connection 

includes a telephone network. 
30 23. The system of claim 14 m which sod tde<xmminication connection 

includes a telephone line. 
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24. The system of claim 14 in which sad speech server consists of a speech 
recognition engine and hardware for connection to a telephone network. 

25. The system of claim 14 in which said coupfing is an electronic circuit. 

26. The system of claim 14 in which said coupfing is a telephone network. 

27. The system of chim 14 in which said coupling is a telephone line, 

28. The system of claim 14 in which said coupling is a fiber optic line. 

29. The system of claim 14 in which said coupling is an electromagnetic wave 
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30. The system of daim 14 in which said network is the Internet 

31. The system of claim 14 in which said speech/web browser is coupled to 
said network such that said speech/web browser executes said functions on said 
network. 

32. The system of claim 3 1 in which said coupling is an electronic circuit 

33. The system of claim 3 1 in which said coupling is a telephone network. 

34. The system of claim 3 1 in which said coupling is a telephone fine. 

35. The system of daim 3 1 in which said coupling is a fiber optic line. 

36. The system ofdaim 31 in which said cou|rfiiuj is an dectrorriagnetic wave 

signal 
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